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There  has  been,  and  is  currently,  no  effective  means  for  conducting  productive  joint  operations 
tests  and  evaluations.  The  fact  that  some  such  efforts  heretofore  have  encountered  difficulties  and 
achieved  few  useful  results  does  not  obviate  the  requirements  for  much  needed  joint  operational  test 
and  evaluation  ( OT&B).  Fitzhugh  Commission,  1970 

Over  the  past  few  deeades,  there  have  been  numerous  eatehphrases  deseribing  visions  for 
reform  in  the  acquisition  community.  Commercial-off-the-Shelf  Procurement;  Cost  as  an 
Independent  Variable,  Simulation-Based  Acquisition,  and  Spiral  Development  are  just  a  few.  The 
latest  such  buzzword  to  hit  the  Department  of  Defense  is'  Transformation.  According  to  General 
Richard  B.  Myers,  Chairman  of  the  Joint  Chiefs  of  Staff,  transformation  deals  more  with  mindsets 
and  processes  than  technologies  and  hardware.  Certainly  there  have  been  numerous  celebrated 
transformational  technologies  over  the  years:  the  machine  gun,  the  tank,  and  aircraft  -  especially 
stealth  -  are  three  prime  examples.  So  how  can  we  incorporate  this  latest  buzzword  into  the 
acquisition  world,  and  more  specifically  into  test  and  evaluation? 

In  this  paper  I'll  offer  some  suggestions  on  how  to  apply  transformation  to  test  and  evaluation 
mindsets  and  processes,  especially  as  they  relate  to  the  joint  world.  I'll  tell  some  "test  stories"  about 
things  I  think  need  to  be  fixed  based  on  my  ten  years  as  an  F-16  test  pilot.  I'll  also  present  some 
broader  examples  from  the  Department  of  Defense  headlines  to  help  support  my  points.  Keep  in 
mind  that  "the  ultimate  objective  of  the  acquisition  process  [is]  to  provide  the  best  and  most  capable 
weapon  systems  to  our  operational  forces."  Although,  the  quote  at  the  top  of  this  page  comes  from 
the  Blue  Ribbon  Defense  Panel  Report  concluded  over  30  years  ago,  unfortunately  there  is  still 
some  truth  to  it  today.  While  we  have  an  established  Joint  Test  and  Evaluation  (JT&E)  program, 
there  are  numerous  intellectual  and  cultural  obstacles  that  must  be  overcome  in  order  to  make  it 
effective  and  efficient.  But  first  a  brief  background  on  JT&E  -is  origin,  set  up,  purpose,  and 


current  projects. 


■TT&R’S  BACKGROUND 


The  Department  of  Defense  (DoD)  Joint  Test  and  Evaluation  program  is  a  direct  outcome  of  the 

1970  Fitzhugh  Commission.  Even  though  the  original  emphasis  was  on  joint  operational  testing 

(OT),  the  program  eventually  expanded  to  also  include  the  developmental  testing  (VT)'community. 

The  Deputy  Director  for  Test  and  Evaluation  within  the  Office  of  the  Secretary  of  Defense  (OSD) 

manages  and  administers  the  program,  which  is  headquartered  in  Alexandria,  VA.  This 

arrangement  establishes  the  OSD  JT&E  program's  independent  status,  aimed  at  minimizing  the 

influence  of  service  bias.  The  program  is  intended  to  accomplish  the  following: 

Assess  the  interoperability  of  service  systems  in  joint  operations  and  explore  potential 
solutions  to  identified  problems. 

•  Evaluate  and  provide  recommendations  for  improvements  in  joint  technical  and  operational 
concepts. 

Validate  testing  methodologies  having  multi-service  application. 

•  Evaluate  technical  performance  of  systems  under  realistic  joint  operational  conditions.  ^ 

DoD  Directive  5000.3-M-4,  .Joint  Test  and  Evaluation  Procedures  Manual,  establishes  the 
policies  and  procedures  for  each  JT&E  program:  It  assigns  a  lead  service  to  conduct  each  project 
and  designates  other  participating  services  to  support  with  personnel  and  resources,  similar  to 
establishing  a  supported  and  the  corresponding  supporting  Combatant  Commanders  during 
military  operations.  As  of  21  February  2003,  there  were  ten  active  JT&E  programs: 

•  Joint  Battle  Damage  Assessment 

Joint  C2  Intelligence,  Surveillance  and  Reconnaissance 

•  Joint  Close  Air  Support 
Joint  Cruise  Missile  Defense 

•  Joint  Global  Positioning  System  Combat  Effectiveness 
Joint  Shipboard  Helicopter  Integration  Process 

Joint  Theater  Distribution 

•  Joint  Methodology  to  Assess  C4ISR  Architecture 
Joint  UAV  For  Time  Sensitive  Operations 

•  Joint  Logistics  /  Planning  Enhancement  ^ 
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Although  the  JT&E  program  has  had  its  successes  (e.g.  the  Joint  Direct  Attack  Munition  and 
the  Joint  Air-to-Surface  Standoff  Missile),  I  don't  think  it  has  yet  reached  its  full  potential.  For 
example,  I've  seen  biases  among  our  services  during  my  decade  of  test  experience.;  I've  even 
worked  on  a  "black"  USAF  missile  program  that  was  classified  simply  because  the  US  Navy  was 
supposed  to  have  the  lead  in  that  particular  arena!  ^  This  goes  decidedly  against  the  whole 
premise  of  the  JT&F  program  as  outlined  above.  But  probably  the  biggest  speed  bump  on  the 
road  to  true  joint  testing  lies  between  the  DT  and  OT  communities,  no  matter  which  service  they 
belong  to.  This  rift  is  where  I  think  transformation  is  needed  the  most. 

TRANSFORMATION  APPI JCATTONS  FOR  .TT&F 

General  Myers  outlines  three  "elements"  of  transformation:  intellectual,  cultural,  and 
technological.  ^  The  first  two  elements  are  probably  the  most  applicable  to  test  and  evaluation 
mindsets  and  processes  since  they  deal  with  how  we  think  and  feel  about  our  line  of  work. 
Although  I'll  offer  some  suggestions  for  each  of  the  three  elements.  I'll  concentrate,  mostly  on 
the  cultural  aspects.  Most  of  my  suggestions  and  those  first  two  of  the  Chairman's  three 
transformational  elements  -  are  therefore  centered  on  people.  And  while  the  term 
"transformation"  implies  geometric  rather  than  incremental  changes  incorporating  revolutionary 
rather  than  evolutionary  ideas,  the  inertia  of  human  nature  makes  such  leaps  and  bounds  difficult 
to  accomplish.  As  a  result,  none  of  my  ideas  are  earth  shattering  or  can  be  easily  implemented 
overnight.  Some  of  them  are  based  on  things  that  have  bothered  me  for  a  long  time,  and  some 
are  the  result  of  rather  recent  events.  My  own  developmental  testing  background  and  service 
affiliation  may  have  even  influenced  some  of  these  ideas,  although  I've  tried  to  keep  any 


resulting  biases  out  of  this  paper. 


The  Intellectual  Element 


General  Myers  stated  that  understanding  transformation  starts  with  the  intellectual  element.' 
People  need  to  know  who  they  are,  what  their  service  is  about,  and  what  is  expected  of  them.  In 
my  case,  I  wasn't  even  aware  a  JT&E  Program  Office  existed  until  I  began  conducting  research 
for  this  paper.  Adding  a  quick  primer  about  this  organization  to  the  various  test  pilot  schools' 
curriculums  would  be  most  beneficial  to  those  of  us  starting  out  in  the  test  and  evaluation 
community.  This  education  could  then  be  expanded  in  the  many  Defense  Acquisition  University 
courses  we're  required  to  take  in  order  to  reach  higher  certification  levels.  Furthermore, 
providing  copies  of  the  JT&E  handbooks  would  be  welcome  and  useful  additions  to  our 
professional  libraries.  Unfortunately,  these  suggestions  only  apply  to  the  DT  community.  Since 
most  operational  testers  are  Weapons  School  graduates,  maybe  incorporating  a  piece  on  how  DT 
and  OT  fit  into  the  overall  test  process  into  that  curriculum  would  also  pay  dividends. 

The  Cultural  Element 

The  Chairman's  second  transformation  pillar  has  numerous  applications  within  the  test  and 
evaluation  community.  The  first  step  in  this  cultural  transformation  should  deal  with  the 
stovepipes  within  and  among  our  military  units  and  services.  There's  nothing  wrong  with 
healthy  competition  among  the  different  specialties,  such  as  friendly  jousting  between  F-15  and 
F-16  fighter  pilots  in  the  baron  a  Friday  night.  The  jokes,  and  ribbing  can  get  downright  brutal 
sometimes,  but  when  duty  calls,  we  re  all  glad  to  be  on  the  same  team.  But  there  is  something 
wrong  with  having  my  credibility  questioned  because  I  wear  a  Test  Pilot  School  Graduate  patch 
on  my  flight  suit  instead  of  a  Weapons  School  Graduate  patch.  I  shouldn't  have  to  be 
accompanied  by  a  "Target  Arm"  when  I'm  giving  a  presentation  on  test  results  affecting  the 
employment  of  a  certain  weapon;  I  shouldn't  need  him  or  her  to  validate  my  briefing  in  order  for 


it  to  be  accepted  by  the  audience.  The  fact  that  I'm  a  professional  in  uniform  should  be 
validation  enough  (not  to  mention  the  fact  that  I  wear  a  2000-hour  F-16  patch  on  my  other  arm!). 

Similarly,  I  shouldn't  need  to  he  accompanied  hy  a  Navy  tester  when  I  give  a  presentation  to 
members  of  my  sister  service.  We  are  working  to  'get  past  these  stovepipes  and  trying  to  be 
more  joint  in  our  endeavors,  but  I  think  we  still  .have  a  ways  to  go.  At  my  last  assignment,  for 
example,  we  had  a  Navy  test  pilot  and  two  Weapons,  School  graduates  attached,  to  our  squadron. 
In  fact,  the  Navy  guy  actually  graduated  from  the  US  Air  Force  Test  Pilot  School!  Additionally, 
we  sent  our  Weapons  School  graduates  to  a  two-week  test  pilot  introduction  course  in  Mojave, 
CA.  This  shows  we're  making  headway  to  overcome  the  stovepipes,  especially  within  the  T&E 
community,  hut  we  still  have  some  work  to  do  with  the  rest  of  the  service  cultures. 

The  Air  Force  culture  easily  recognizes  the  prestige  of  our  Weapons  School  and  its 
graduates;  it  doesn't  readily  acknowledge  the  "Golden  Arms"  of  the  test  and  evaluation 
community.  In  fact,  the  testers  are  often  saddled  with  the  hlame  for  program  schedule  slips  and 
cost  overruns  when  technical  difficulties  arise.  Historically,  according  to  the  Honorable  Thomas 
P.  Christie,  Director  of  Operational  Test  and  Evaluation,  "When  these  technical  problems 
surfaced,  primarily  in  development  testing,  the  stretch-out  and  cost  growth  cycle  began.  We 
found  that  the  messenger  of  bad  tidings,  most  often  'the  T&E  community,  was  then  tarred  with 
responsibility  for  the  stretch  outs."  *  This  is  a  classic  case  of  shooting  the  messenger  rather  than 
trying  to  solve  the  root  of  the  problem. 

Another  way  to  tackle  the  stovepipes  between  the  DT  and  OT  communities  is  to  work 
together  on  test  programs.  Eglin  AFB,  EL,  provides  such  an  opportunity  by  having  a  squadron 
of  developmental  testers  located  adjacent  to  an  operational  test  squadron.  One  of  the  programs  I 
worked  on  while  stationed  there  in  the  mid-1990s  was  the  High-speed  Anti-Radiation  Missile 
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(HARM)  Targeting  System  (HTS).  A  combined  team  from  the  DT  and  OT  communities, 
"Golden  Arms"  and  "Target  Arms"  alike,  deployed  to  Edwards  AFB,  CA,  for  a  series  of 
combined  DT/OT  testing  of  the  HTS.  We  flew  together  in  the  same  4-ship  flights,  even  rode  in 
each  other's  two-seat  F-16s,  recording  both  quantitative  and  qualitative  data.  We  were  able  to 
see  how  "the  other  side"  did  business  and  gained  an  appreciation  for  it.  It's  not  unnatural  to 
criticize  something  that's  unfamiliar;  this  program  served  to  break  that  barrier,  even  if  it  was  for 
just.'  the  small  handful  of  us.  True,  this  is  but  one  minor  example,  but  one  with  positive  lessons 
that  shouldn't  have  to  be  re-learned  throughout  the  test  and  evaluation  community.  There  is  a 
golden  opportunity  to  take  advantage  of  this  concept  on  the  western  US  test  ranges.  Operational 
testers  from  Nellis  AFB,  NV,  and  developmental  testers  from  either  Edwards  AFB  or  China 
Lake  Naval  Air  Station,  CA,  could  easily  meet  for  combined  testing  on  either  of  the  two  ranges. 
It's  a  shame  we  don't. 

Let's  turn  now  from  the  joint  to,  the  Joint  arena,  that  is,  from  cooperation  among  the  services 
to  endeavors  that  may  also  include  coalition  partners  and  civilian  contractors  and  agencies.  One 
of  the  biggest  cultural  dilemmas  in  this  arena  deals  with  fielding  systems  before  they  are 
thoroughly  tested.  There  are  some  success  stories,  such  as  JSTARS  during  the  first  Gulf  War. 
Unfortunately  there  are  also  some  failures.  Tragically  some  of  these  have  cost  people  their  lives. 

The  Bradley  Fighting  Vehicle  is  one  example  of  a  "buy  now,  fix  later"  philosophy  that  was 
so  prevalent  during  the  1960s,  1970s,  and  early  1980s.  The  Bradley  is  an  armored  personnel 
carrier  designed  to  replace  the  M-1 13,  whose  armor  wasn't  thick  enough  to  protect  its  troops 
from  anything  larger  than  small  arms  fire.  Although  the  Bradley  was  first  deployed  in  1983,  it 
wasn't  live-fire  tested  until  1985,  at  which  point  it  was  found  to'  be  highly  vulnerable  to  anti- 
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armor  weapons.  Additionally,  the  "unsinkable"  Bradley  wasn't!  After  the  11  Bradley 


swamped  at  Fort  Benning,  GA,  the  US  Army  suspended  operations  until  the  problem  could  be 
corrected.  ^  The  saga  was  even  documented  in  a  made-for-HBO  television  comedy  (based  on  a 
true  story,  of  course)  titled  The  Pentagon  Wars  in  1998.  Fortunately  these  incidents  did  not 
result  in  any  loss  of  life  due  to  the  rush  to  field  the  system  before  it  was  thoroughly  tested  for 
effectiveness  and  suitability.  Such  is  not  the  case  for  my  next  example. 

The  Air  Force's  T-3  Firefly  initial  trainer  aircraft  offers  another  illustration  of  what  can 
happen  when  a  system  is  not  properly  tested  prior  to  being  fielded.  In  1995,  the,  US  Air  Force 
Academy  replaced  its  aging  T-41  fleet  with  the  new  T-3  The  T-41  had  an  impeccable  safety, 
record:  in  over  30  years  of  dedicated  service,  there  had  never  been  a  single  fatality.  In  the  first 
28  months  of  flying  T-3s,  however,  there  were  three  crashes  and  six  fatalities.  The  first  occurred 
on  22  February  1995  during  spin  training.  The  second  happened  on  30  September  1996  when 
the  single  propeller-driven  engine  quit  in  the  landing  pattern.  The  final  accident  took  place  on  25 
June  1997,  again  when  the  engine  failed,  this  time  at  low  altitude.  The  plane  was  finally 
permanently  grounded  a  month  later,  when  another  aircrew  experienced  engine  failure,  this  time 
(fortunately)  overhead  the  airfield  and  in  a  position  to  dead-stick  the  aircraft  to  the  runway." 
Investigators  discovered  that  the  aircraft  had  not  been  properly  tested  at  the  higher  absolute  and 
lower  density  altitudes  of  the  Academy  airfield  (field  elevation  of  6,572  feet)  which  could  help 
explain  the  aircraft's  poor  performance  during  spin  recoveries  and  its  susceptibility  to  engine 
malfunctions  during  control  setting  changes. 

These  two  examples  highlight  the  cultural  dilemma  surrounding  the  pressures  to  field  a. 
system  before  it  is  ready.  Sometimes  that  pressure  comes  from  senior  service  leadership; 
sometimes  it  is  self-imposed  by  the  contractor  supplying  the  system.  The  common  sense 
suggestion  is  to  not  do  it;  why  risk  human  lives  just  to  meet  a  deadline  or  milestone?  I  realize 


that  there  is  much  more  to  both  of  these  stories.  My  intent  in  using  them  is  to  point  out  the 
problem  with  not  fully  appreciating  technical  challenges  that  may  face  a  program  from  the 
beginning.  Simply  sweeping  them  under,  the  proverbial  rug  can  have  severe  consequences.  We 
in  the  T&E  community  must  therefore  be  prepared  to  call,  for  a  timeout  and  address  these 
challenges  rather  than  overlooking  them  in  the  hope  they'll  go  away. 

The  multi-service  V-22  Osprey  provides  another  example  of  a  cultural  transformation  that 
needs  to  take  place  within  the  test  and  evaluation  environment.  The  tilt-rotor  aircraft  program 
actually  began  way  back  in  the  early  1980s  '  as'  a  replacement  for  the  CH=46  troop  transport 
helicopters.  Three  of  the  first  15  V-22s  delivered  were  involved  in  fatal  crashes,  killing  a  total  of 
30  Marines  since  tests  began  in  1986.  The  most  horrific  of  these  was  the  8  April  2000  crash  in 
Arizona  that  killed  all  19  Marines  onboard.  The  most  recent  crash  was  on  1 1  December  2000  in 


North  Carolina,  where  the  crew  of  four  perished.  Among  them  was  the  pilot,  a  Marine  Corps 
Lieutenant  Colonel  who  was  chief  of  the  Osprey  testing  program  and  slated  to  become 
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commander  of  the  first  operational  squadron.  Although  the  April  crash  was  blamed  on  pilot 
error,  the  December  crash  which  killed  the  most  experienced  Osprey  pilot  in  the  program  was 
caused  in  part  by  a  hydraulic  failure.  According  to  the  Osprey  Program  Manager,  USMC 
Colonel  Nolan  Schmidt,  in  his  testimony  to  the  US  Senate  Armed  Services  Committee  in  April 
2001,  there  had  been  earlier  catastrophic  hydraulic  failures  similar  to  the  one  that  led  to  the 
North  Carolina  crash.  The  most  recent  failure  occurred  during  a  February  2000  operations 
evaluation  (OPEVAL),  information  that  Schmidt  hadn't  received  until  a'  week  prior  to  his 


testimony.  Schmidt  stated  the  "lack  of  effective  communications  ...  was  an  outgrowth  of 
deliberately  separate  test  and  program  office  structures."  He  further  charged  that  "contact 


between  OPEVAL  and  the  program  office  is  a  'deviation  from  the  norm. 
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'  This  case  highlights  the  cultural  dilemma  albeit  in  a  very  simplified  manner  -  of  whether  or 
not  to  be  the  bearer  of  bad  news.  There  is  credence  to  the  argument  of  not  wanting  to  overload  "the 
loss"  with  trivial  problems  that  can  be  solved  at  a  lower  level,  and  within  a  small  organizatiop  that 
may  be  easy  to  do.  But  when  it  comes  to  a  large  acquisition  program,  where  the  corisequences  are 
often  measured  in  human  lives,  this  dilemma  becomes  anything  but  trivial. 

The  V-22  case  also  highlights  the  need  to  get  government  testers  involved  early  in  the  process. 
Having  the  contractor  do  the  testing  is  like  kissing  your  sister.  Early  involvement  also  injects  a 
feeling  of  teamwork  rather  than  an  us-versus-them  mentality.  Integrated  Product  Teams  consisting 
of  contractors  and  government  representatives  help  overcome  this  cultural  dilemma. 

An  important  point  to  emphasize  here  is  the  absolute  requirement  for  transformation,  buy-in  at 
the  highest  levels  of  leadership.  If  these  cultural  changes  are  not  totally  espoused  by  our  most 
senior  leaders  right  from  the  start,  in  the  different  services  and  in  the  different  T&E  communities, 
then  the  ideas  will  once  again  die  on  the  vine.  Leaders  must  "walk  the  talk"  if  there  is  any  hope  of 
accomplishing  any  of  these  cultural  transformations.  As  Mr.  Christie  stated,  "We've  never  really 
taken,  [these  reform  efforts]  to  heart;  granted,  we  ...continue  to  make  changes  to  the  process,  but  we 
have  yet  to  really  come  to  grips  with  some  of  the  root  causes." '  ^  The  Fitzhugh  Commission  tried  to 
push  a  change  to  the  process,  but  it  seems  to  me  it  was  about  as  effective  as  pushing  a  wet  noodle. 
That  noodle  will  only  be  effective  if  it  gets  pulled  from  the  top. 

The  Technological  Element 

The  third  and  final  element  of  the  Chairman's  transformation  vision  deals  with  technology, 
which  in  the  test  and  evaluation  world  translates  to  infrastructure.  Technologies  such  as  video 
teleconferencing  and  speakerphones  allow  real-time  briefings  and  debriefings,  making  it  easier  for 
units  at  different  locations  (such  as  Nellis  and  Edwards)  to  test  together.  But  as  the  systems  under 
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test  become  increasingly  more  complex,  the  required  data  rates,  and  volumes  also  multiply. 
Telemetry  coverage  and  quality  therefore  become  increasingly  important,  which  further  results  in 
bandwidth  limitations  and  frequency  management  problems.  Dealing  with  these  challenges 
requires  fostering  relationships  with  civilian  organizations  such  as  the  FAA,  thereby  driving  the 
requirement  for  Joint  testing  to  be  in  fact  spelled  with  a  capital  J. 

CONCLUSION 

The  idea  of  acquisition,  reform  is  not  new.  Neither  are'  most  of  the  suggestions  1  offer  on  how,  to 
make  the  test  and  evaluation  community  more  efficient  and  effective.  What  is  new  is  this  idea  of 
transformation,  of  changing  how  we  think  and,  feel  about  the  work  we  do  as  testers.  The  21" 
Century  has  certainly  brought  on  a  new  set  of  challenges:  rapid  technology  advances,  less  well- 
defined  threats,  and  concerns  over  homeland  security.  As  Mr.  Christie  put  it,  "We  [the  T&E 
community]  must  seriously  consider  restructuring  our  thinking  to  better  meet  the  new  challenges. 

We  can  begin  this  transformation  by  educating  our  DT.  and  OT  communities  in  each  other's 
requirements  and  ways  of  doing  business.  We  need,  to  break  down  the  stovepipes  within  and  among 
the  services.  We  must  ensure  our  systems  are  sufficiently  tested  before  being  fielded.  We  shouldn't 
be  afraid  to  speak  up  when  a  problem  arises.  We  need  to  be  involved  early  in  the  test  process.  And 
we  need  buy-in  from  our  senior  leaders.  Again,  our  ultimate  responsibility  as  testers  is  to  figure  out 
"how  to  get  more  effective  and  suitable  systems  into  the  hands  of  our  operational  combat  forces 
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more  rapidly  and  at  less  costs  to  the  taxpayers."  Maybe  this  latest  entry  onto  the  storied  list  of 
buzzwords  and  catchphrases  -  transformation  -  will  actually  be  the  catalyst  we've  been  waiting  for 
to  reform  our  joint  test  and  evaluation  mindsets  and  processes.  We  can  only  hope  so. 
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